User detection and tracking, such as head and facial detection and tracking, has several important applications for users of computing devices, such as personal computers, tablets, smartphones, or head-mounted displays and other wearable computing devices (e.g., eyeglasses, visors, gloves, watches, wristbands, etc.). User detection and tracking can enable users to interact with computing devices in new and interesting ways. For example, a computing device can be configured to recognize certain head and facial gestures, such as head nods or rolls, eye winks or other ocular motion, or lip movement as input to the device to perform tasks upon recognition of the gestures. User detection and tracking can also streamline video conferencing by detecting, tracking, and transmitting user movement (e.g., facial expressions, hand waves) and rendering video from models of the video conference participants and the tracked user movement instead of capturing and streaming video. Virtual and augmented reality applications can also depend on head pose estimation to correctly render image or video data according to the current perspective of the user. Gaze estimation can be enhanced when the pose of the user's head is known. Speech recognition can be made more robust by monitoring user lip movement in conjunction with analysis of audio data. Certain three-dimensional (3-D) or stereographic displays may also depend upon head pose and/or gaze estimation. Thus, accurate user detection and tracking may be critical to a variety of functions of a computing device. User detection, however, can be challenging because of variations in appearance, scale, rotation, position, and orientation. Other factors such as camera characteristics, illumination conditions, and occlusion can also affect how a user is represented in an image. Thus, even the most accurate user detection and tracking algorithms may not be as precise as desired.